Skip to content

fix: unwrap response envelopes + correct Pydantic field names for v0.1.1#6

Merged
OussemaFr merged 2 commits into
mainfrom
fix/envelope-and-typed-models
Jun 22, 2026
Merged

fix: unwrap response envelopes + correct Pydantic field names for v0.1.1#6
OussemaFr merged 2 commits into
mainfrom
fix/envelope-and-typed-models

Conversation

@OussemaFr

Copy link
Copy Markdown
Member

What was broken in v0.1.0

The three typed-model methods (`get_page_info`, `get_profile_details`, `get_group_details`) returned Pydantic models with no populated fields against the real API. Two root causes, both discovered via PR #5's smoke test:

1. Envelope shape is inconsistent across endpoints

Endpoint Real envelope
`/facebook/pages/details` payload under key `"0"` (a literal string)
`/instagram/profile/details` payload under `"data"` (with sibling `"success"`, `"message"`, `"meta"`)
`/facebook/groups/details` NO wrapper — payload spread at top level

v0.1.0 passed the whole envelope to `Model.model_validate(...)` for all three, so the typed models matched nothing.

2. Pydantic field names were guessed (wrongly)

I built v0.1.0 from `apiSources.ts` (the endpoint catalog) without ever seeing a real API response. The real field names differ:

Model v0.1.0 guess Real API
PageInfo `id, name, likes, verified, about` `ad_page_id, title, likes_count, is_business_page_active, bio`
ProfileInfo `followers, posts_count` `followers_count, media_count`
GroupInfo `member_count, privacy` `group_member_count, privacy_info_text`

Fixes

Code

  • `get_page_info` (sync + async) → unwrap `body["0"]` before validate
  • `get_profile_details` (sync + async) → unwrap `body["data"]` before validate
  • `get_group_details` → no code change (no wrapper to unwrap)

Both unwrappers fall back to the raw body if the envelope shape ever changes upstream.

Models (all rewritten with real field names verified against the live API today)

  • `PageInfo`: ad_page_id, user_id, title, url, category, bio, description, address, phone, email, website, followers_count, likes_count, image, rating, business_hours, twitter, linkedin, instagram, pinterest, telegram, youtube
  • `GroupInfo`: group_id, group_member_count, description_text, created_time, group_rules, group_history, admin_tags, group_locations, number_of_posts_in_last_day, number_of_posts_in_last_month, privacy_info_text
  • `ProfileInfo`: id, pk, fbid, username, full_name, biography, followers_count, following_count, media_count, total_clips_count, highlight_reel_count, is_verified, is_private, is_business_account, profile_pic_url, profile_pic_url_hd, external_url, has_clips, has_guides, account_type, category_name, business_email, business_phone_number, etc.

Tests — mock fixtures now mirror the REAL envelope shape, so the mocked tests actually validate the real codepath. Assertions updated to use real field names.

Version: 0.1.0 → 0.1.1 + CHANGELOG entry with breaking-change callout.

Verification

Local sanity check (all green):

  • `ruff check .` ✅
  • `ruff format --check .` ✅
  • `pytest` → 33 passed, 80% coverage
  • Real-API smoke test → `instagram.get_profile_details` populates 47 typed fields (was 0 in v0.1.0). Examples: `followers_count=685000000`, `media_count=1`, `is_verified=True`, `username='instagram'`.

What does NOT change

  • Public import paths (`from socialapis import Facebook, Instagram, ...`)
  • Wire-level behavior (URLs, headers, query params)
  • Typed exception hierarchy
  • Migration aliases (`FacebookScraper`, `InstagramScraper`)
  • All `dict[str, Any]`-returning methods (`get_page_posts`, `search_*`, marketplace, etc.) — unaffected

Note on smoke test

PR #5 (`chore/integration-smoke-test`) ships the script. After this PR merges, the smoke script needs ONE small assertion update (read `page.ad_page_id` instead of `page.id`). I'll push that as a follow-up commit to PR #5's branch.

After this merges → release v0.1.1

```bash
git checkout main && git pull
git tag v0.1.1
git push origin v0.1.1
```

Release workflow auto-publishes `socialapis-sdk==0.1.1` to PyPI via the existing Trusted Publisher. The badge updates automatically.

v0.1.0 shipped with three typed-model methods (get_page_info,
get_group_details, get_profile_details) that returned models with
no populated fields against the real API. Two root causes, both
discovered via the integration smoke test:

1. Envelope shape is inconsistent across endpoints (the SDK passed
   the whole envelope to model_validate instead of unwrapping the
   payload first):

   FB /facebook/pages/details   payload under key "0" (string!)
   IG /instagram/profile/details payload under "data"
   FB /facebook/groups/details   NO wrapper (payload at top level)

2. The v0.1.0 Pydantic field names were invented based on what I
   guessed the API would return. Reality:

   PageInfo:     guessed `id, name, likes, verified, about`
                 real:    `ad_page_id, title, likes_count,
                          is_business_page_active, bio`

   ProfileInfo:  guessed `followers, posts_count`
                 real:    `followers_count, media_count`

   GroupInfo:    guessed `member_count, privacy`
                 real:    `group_member_count, privacy_info_text`

Fixes
=====

Code:
- socialapis/facebook/_client.py: get_page_info (sync + async) now
  unwraps body["0"] before model_validate. Falls back to the raw
  body if the shape ever changes upstream.
- socialapis/instagram/_client.py: get_profile_details (sync +
  async) now unwraps body["data"] before model_validate. Same
  fallback.
- (No change needed for get_group_details — that endpoint has no
  wrapper; the raw body already matches the model.)

Models (all rewritten with real field names verified against the
live API, 2026-06-22):
- socialapis/facebook/_types.py: PageInfo + GroupInfo
- socialapis/instagram/_types.py: ProfileInfo

Tests:
- tests/test_facebook.py: SAMPLE_PAGE_INFO now mirrors the real
  envelope shape ({"0": {...}, "message", "meta"}). Assertions
  updated to use real field names (page.title, page.likes_count,
  page.followers_count, page.image).
- tests/test_instagram.py: SAMPLE_PROFILE now mirrors the real
  envelope ({"success", "data": {...}, "message", "meta"}).
  Assertions updated (profile.followers_count, profile.media_count).

Version: 0.1.0 → 0.1.1, CHANGELOG updated with breaking-change
callout for users who were reading the (always-None) attributes on
the typed models in v0.1.0.

Verification
============

  ruff check . + ruff format --check .  → all green
  pytest                                 → 33 passed, 80% coverage
  integration_smoke.py against real API → IG profile populates 47
                                          fields including followers_count
                                          (685M), media_count, is_verified

Backwards compat
================

Everything except the renamed typed-model attributes is preserved:
- Public imports (`from socialapis import ...`)
- Wire-level behavior (URLs, headers, params)
- Exception hierarchy (AuthenticationError, RateLimitError, ...)
- Migration aliases (FacebookScraper, InstagramScraper)
- All dict[str, Any]-returning methods are unaffected

The dict-returning methods (get_page_posts, search_*, marketplace_*,
etc.) still pass through the raw envelope unchanged. Users who want
the inner payload can do `result["data"]` themselves. We'll consider
auto-unwrap in a future release once more endpoint-specific behavior
is verified.
OussemaFr added a commit that referenced this pull request Jun 22, 2026
The smoke test reads `page.id` and probes `BadRequestError` with a
nonexistent slug. Both assumptions need adjusting after the typed-
model rewrite in #6:

- PageInfo now uses `ad_page_id` (and `user_id`) — the API doesn't
  return a bare `id` field for pages. Updated the assertion.
- The API returns 200 with an empty payload for nonexistent page
  slugs, not a 4xx. To still validate the SDK's error mapping
  works against a real 4xx, the test now sends a deliberately
  malformed request (no params) which the API rejects with 400.

Stays self-contained — no new dependencies, no changes outside
scripts/integration_smoke.py.
OussemaFr added a commit that referenced this pull request Jun 22, 2026
* chore: add integration smoke test script + harden .gitignore

Adds scripts/integration_smoke.py for local validation against the
live socialapis.io REST API. The script makes one call per major
endpoint category (Facebook pages/groups/search/ads/marketplace,
Instagram profile/posts/search/reels, Account usage, plus two
error-mapping checks) and reports per-method pass/fail.

Why a script instead of CI tests:
- Real API tests need a token, which means either a secret in CI
  (leak surface) or manual-trigger workflows (low ROI)
- The mocked tests already cover wire-level behavior in CI
- A local smoke test catches the bugs mocks can't: wrong endpoint
  paths, Pydantic field name mismatches, envelope shape drift
- One run is enough; this is a "before shipping a big change"
  ritual, not continuous

The script reads SOCIALAPIS_TOKEN from env, never prints the value,
never persists it. It's gitignored-by-association via the .gitignore
additions for .env/.token files.

Also hardens .gitignore with defense-in-depth entries for env files
and tokens (.env, .env.*, *.token, .socialapis_token) so an accidental
commit doesn't land a secret in public history.

What the first run found (validation context, fix in a follow-up PR):
- All 12 endpoint calls work at the wire level — auth, routing,
  forwarding kwargs all good
- AuthenticationError mapping works correctly (bad token → 401)
- But the typed-model methods (get_page_info, get_profile_details,
  get_group_details) all return Pydantic models with no populated
  fields, because the real API:
    a) Wraps responses in inconsistent envelopes (key "0" for FB
       pages, "data" for IG profiles, no wrapper at all for FB
       groups)
    b) Uses different field names than my models guessed
       (likes_count not likes, followers_count not followers,
       media_count not posts_count, etc.)

Follow-up PR will fix the envelope extractor + correct field names,
then bump to v0.1.1.

* chore: match smoke test assertions to the v0.1.1 model field names

The smoke test reads `page.id` and probes `BadRequestError` with a
nonexistent slug. Both assumptions need adjusting after the typed-
model rewrite in #6:

- PageInfo now uses `ad_page_id` (and `user_id`) — the API doesn't
  return a bare `id` field for pages. Updated the assertion.
- The API returns 200 with an empty payload for nonexistent page
  slugs, not a 4xx. To still validate the SDK's error mapping
  works against a real 4xx, the test now sends a deliberately
  malformed request (no params) which the API rejects with 400.

Stays self-contained — no new dependencies, no changes outside
scripts/integration_smoke.py.
mypy --strict required type arguments on the bare list / dict
annotations I added for GroupInfo.privacy_info_text, group_rules,
group_history, admin_tags, group_locations and ProfileInfo.pronouns,
account_badges. Replaced with list[Any] / dict[str, Any], imported
Any in both files.

Pure type-annotation tightening — no behavior change, all 33 tests
still pass.
@OussemaFr OussemaFr merged commit 44ee601 into main Jun 22, 2026
6 checks passed
OussemaFr added a commit that referenced this pull request Jun 22, 2026
v0.1.0's README and examples used the field names I'd guessed for
the typed models. v0.1.1 (PR #6) rewrote the models with the real
field names the API returns. The README + two example scripts
still referenced the old names — anyone copy-pasting the code
would get AttributeError.

Field-name fixes (applied to README + examples/quickstart.py +
examples/migrate-from-kevinzg.py):

  page.name        → page.title
  page.likes       → page.likes_count
  page.followers   → page.followers_count
  page.about       → page.bio
  page.verified    → dropped (no equivalent in real API response)
  profile.followers   → profile.followers_count
  profile.posts_count → profile.media_count

profile.full_name and profile.is_verified were already correct.

Verified locally:
  ruff check .          → All checks passed!
  ruff format --check . → 17 files already formatted
  pytest                → 33 passed, 80% coverage

No code changes to the SDK — pure docs / example sync.
@OussemaFr OussemaFr deleted the fix/envelope-and-typed-models branch June 22, 2026 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant